fetch_util 0.3.0 → 0.3.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (24) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +37 -0
  3. data/Rakefile +5 -0
  4. data/lib/fetch_util/assets/extract.js +1 -1
  5. data/lib/fetch_util/browser/interaction_helpers/consent_helpers.rb +187 -104
  6. data/lib/fetch_util/browser/interaction_helpers/dom_interaction.rb +14 -2
  7. data/lib/fetch_util/browser/navigation/headers_and_readiness.rb +0 -7
  8. data/lib/fetch_util/browser/navigation/navigator_patch.rb +3 -1
  9. data/lib/fetch_util/browser/site_stabilization/community_and_marketplace.rb +28 -27
  10. data/lib/fetch_util/browser/site_stabilization/gitlab_repo.rb +32 -0
  11. data/lib/fetch_util/browser/site_stabilization/social_platforms.rb +25 -32
  12. data/lib/fetch_util/browser/site_stabilization.rb +2 -0
  13. data/lib/fetch_util/browser/stabilization/page_flow.rb +93 -10
  14. data/lib/fetch_util/browser.rb +55 -16
  15. data/lib/fetch_util/cli.rb +1 -0
  16. data/lib/fetch_util/extractor.rb +27 -3
  17. data/lib/fetch_util/fetcher.rb +696 -67
  18. data/lib/fetch_util/parallel_fetcher.rb +15 -20
  19. data/lib/fetch_util/raw_docs_fallback.rb +48 -24
  20. data/lib/fetch_util/regulatory/http_client.rb +111 -43
  21. data/lib/fetch_util/result.rb +100 -3
  22. data/lib/fetch_util/version.rb +1 -1
  23. data/lib/fetch_util.rb +12 -2
  24. metadata +3 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: fa6e99e3ed511c2cab332a2914dbf37ffce3e248bd2ba08b967ef744c672b806
4
- data.tar.gz: 2e4a8d928de0a9f4b2d60d81373326743341fcc0454cd087a4a7ced4f52393bd
3
+ metadata.gz: 8f8adcce7137bb744a361c30f0a2f47ea9f26e751cab745f0b298148151c0ea8
4
+ data.tar.gz: edbff86bc67288848d4e4c2548c6e8c5b7b53bcc4c29cc10a3d70e9831d19dec
5
5
  SHA512:
6
- metadata.gz: c0dc535d27ef97d4a6ee65e154f9c3503ed78ec5df93280e6ee14a90d1c83ec641a4a53e78419056718d9cd78480c8124cf132cc06854140296df481e7153e42
7
- data.tar.gz: 54303e5889abeacb1341602125eb90a8e1e38a054b3e6a853bd0789e3070272c86f9e8058b88f2282c3734c9c63af8cee4ffa5cd56285aa49c8d25eee6068131
6
+ metadata.gz: 85ce1b2472bb679c4f5941d3f828cfbc6455732b81c31952e1d9a5582fbfaa78af247e3ea88fe7a20fdfdcd60d6f50b124e662d6f139891c703060938ff4977d
7
+ data.tar.gz: b43a3c4ac0ab6d469c895ca8a12aca159b9d8112de63074497dec4e73178340b2bab666c5f2c7bb3a7ac9f1326c551816a16fed65733f06449a95bab1178a150
data/CHANGELOG.md CHANGED
@@ -2,6 +2,43 @@
2
2
 
3
3
  ## Unreleased
4
4
 
5
+ ## v0.3.2 - 2026-07-09
6
+
7
+ ### Added
8
+
9
+ - Add dedicated extraction coverage for public Chosun, Xinhua weekly indexes, O Pais, Avesta, Walla, Jang, 20minutos, NHK, Protothema, Lenta, and YNA article/list pages.
10
+ - Add a Hurriyet homepage regression fixture to protect non-English homepage extraction.
11
+
12
+ ### Fixed
13
+
14
+ - Normalize non-ASCII IRI input URLs before browser navigation.
15
+ - Harden fetch resilience by retrying transient navigation and pending-connection failures.
16
+ - Improve generic article extraction and MediaWiki/Wikipedia handling while preserving localized truncation and warning semantics.
17
+ - Reduce false positives for homepage index mismatches, stale content, script-language mismatches, short news CTA multi-topic pages, transliterated slugs, and long non-English articles.
18
+ - Preserve article content through video metadata, France24 DOM stabilization, and short-description-root edge cases.
19
+
20
+ ### Changed
21
+
22
+ - Consolidate list extraction helpers and reduce duplicated extraction/profile code across generic, WordPress, simple-profile, and job-list paths.
23
+ - Refine generic extraction and non-Latin spec style without changing the public API.
24
+
25
+ ### Performance
26
+
27
+ - Speed up browser stabilization and Ruby fetch runtime.
28
+ - Share browser pools, avoid unnecessary asset rebuilds in specs, optimize asset checks/readable text, and streamline extractor integration specs.
29
+
30
+ ## v0.3.1 - 2026-07-06
31
+
32
+ - Improve primary content-root selection across long documents, article pages, docs, Drupal and institutional pages, publisher abstracts, Substack, Discourse, Trope Wiki, PLOS, Elsevier, ACS, HighWire, and legal/statute pages so body content is preferred over chrome, teasers, and related rails.
33
+ - Expand list, index, card, and hub classification for section feeds, job results, government and legal portals, legal contents/search pages, institutional case lists, AEM/Drupal/custom CMS cards, commerce product grids, Statuspage boards, standards records, package registries, and public Mastodon timelines.
34
+ - Add or improve extraction for structured reference and scholarly records including arXiv abstracts, dictionary definitions, GitLab repository READMEs, Cornell Wex articles, legal conventions, US Code provisions, OEIS/RCSB/NIST records, NCBI/Ensembl gene records, IUCN species assessments, IEEE Xplore abstracts, and official statute text.
35
+ - Reduce false positive warnings for stale content, reposts, consent/paywall walls, Polish language mismatches, docs pages, subscriptions, scientific/legal/institutional pages, legal cases, DOI/publisher redirects, same-organization redirects, statutes, and treaty indexes.
36
+ - Improve error and interstitial handling for auth redirects, PDFs, dominant unavailable pages, soft errors, press-hold challenges, dead DOI redirects, legal soft 404s, publisher unavailable pages, not-found pages, nav-only records, short landing heroes, and privacy fragments.
37
+ - Clean markdown artifacts from malformed card grids, legal citations, table-based content, nested tables, treaty collections, article promo widgets, reference rails, undefined image text, and repeated list card details.
38
+ - Ensure browser extraction assets are rebuilt during verify, spec, build, and release paths; guard extraction asset manifest completeness.
39
+ - Consolidate shared extraction helpers for CMS component hubs, scholarly articles, platform profiles, repository READMEs, card lists, not-found/interstitial detection, legal text, table markdown, cleanup vocabulary, result field mapping, profile HTML/articles, Chrome stripping, platform signatures, list item results, and spec assertions.
40
+ - Ignore local Serena project metadata from Git.
41
+
5
42
  ## v0.3.0 - 2026-06-21
6
43
 
7
44
  - Add generic portal, marketplace, and booking homepage root selection so lead-story lists are extracted from shell pages the previous docs, repo, and community dispatchers missed; already-handled homepages are unaffected.
data/Rakefile CHANGED
@@ -15,4 +15,9 @@ task :verify_extract_assets do
15
15
  ruby "script/build_extract_assets.rb", "--check"
16
16
  end
17
17
 
18
+ task verify_extract_assets: :build_extract_assets
19
+ task spec: :build_extract_assets
20
+ task build: :build_extract_assets
21
+ task release: :build_extract_assets
22
+
18
23
  task default: %i[verify_extract_assets spec rubocop]